Repository navigation
FEAT: Prompt generation jobs in the backend (-5) - #4134
Richard Lundeen (richlundeen) merged 57 commits into
Conversation
Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical storage and cross-backend coordination races can overwrite terminal state, hide jobs, and release dataset-name reservations while original workers remain active, allowing duplicate generation.
5 open issues
Reject cancellation of remotely owned active jobs · New Reserve dataset names atomically across shared-storage backends · New Drain cancelled checkpoint writes before releasing the service lock · New Serialize cache initialization with generation mutations · New Persist and restore generation requests for objective datasets · New
What changed in this PR
This PR adds backend-managed prompt generation jobs so clients can draft, estimate, review, resume, and save generated datasets using PyRIT’s existing generators and dataset providers.
Changes:
- Integrates generation jobs with backend routing, runtime lifecycle, estimates, and dataset packaging.
- Bundles ASSERT behavior and suite presets with attribution.
- Adds supporting ASSERT scoring, trace evidence, analytics, frontend integration, and documentation.
| File | Description |
|---|---|
| THIRD_PARTY_NOTICES.txt | Updates ASSERT attribution. |
| tests/unit/executor/promptgen/test_target_objective_generator.py | Tests generation estimates. |
| tests/unit/datasets/test_target_objective_provider.py | Tests result packaging. |
| tests/unit/common/test_lazy_package_imports.py | Updates analytics export count. |
| tests/unit/assert_mocks.py | Adds checkpoint fixtures. |
| tests/unit/analytics/test_rate_statistics.py | Tests binary and macro rates. |
| pyrit/score/true_false/true_false_fallback_scorer.py | Adds configurable fallback triggers. |
| pyrit/score/scorer.py | Identifies missing-evidence results. |
| pyrit/score/observation/otel_span_exporter.py | Exports richer trace evidence. |
| pyrit/score/observation/message_trace_resolver.py | Resolves conversation trace links. |
| pyrit/score/observation/execution.py | Shares evidence acquisition across scorers. |
| pyrit/score/observation/__init__.py | Exports trace observation APIs. |
| pyrit/score/float_scale/float_scale_fallback_scorer.py | Adds configurable fallback triggers. |
| pyrit/score/__init__.py | Exports ASSERT and tracing APIs. |
| pyrit/scenario/core/scenario.py | Extracts scenario identity parameters. |
| pyrit/prompt_target/common/target_trace_config.py | Adds conversation identifiers to spans. |
| pyrit/prompt_target/common/prompt_target.py | Exposes target tracing configuration. |
| pyrit/models/score/__init__.py | Exports execution-trace models. |
| pyrit/models/prompt_generation_estimate.py | Defines generation estimate contracts. |
| pyrit/models/analytics.py | Adds rate result models. |
| pyrit/executor/promptgen/target_objective_generator.py | Estimates objective-generation work. |
| pyrit/executor/promptgen/__init__.py | Exports objective-generation results. |
| pyrit/datasets/seed_datasets/generated/target_objective_provider.py | Packages existing generation results. |
| pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml | Guides execution-evidence judgments. |
| pyrit/datasets/score/assert/judge_presets/tool-use.yaml | Adds tool-use judging criteria. |
| pyrit/datasets/score/assert/judge_presets/safety-extended.yaml | Adds extended safety criteria. |
| pyrit/datasets/score/assert/judge_presets/safety-core.yaml | References core safety dimensions. |
| pyrit/datasets/score/assert/judge_presets/robustness.yaml | Adds robustness judging criteria. |
| pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml | Adds policy-adherence criteria. |
| pyrit/datasets/score/assert/judge_presets/operational.yaml | Adds operational judging criteria. |
| pyrit/datasets/score/assert/judge_presets/multi-turn.yaml | Adds multi-turn judging criteria. |
| pyrit/datasets/score/assert/judge_presets/instruction-following.yaml | Adds instruction-following criteria. |
| pyrit/datasets/score/assert/judge_presets/grounding.yaml | Adds grounding judging criteria. |
| pyrit/datasets/score/assert/judge_presets/communication.yaml | Adds communication judging criteria. |
| pyrit/datasets/score/assert/judge_presets/alignment.yaml | Adds alignment judging criteria. |
| pyrit/datasets/score/assert/dimension_system_prompt.yaml | Adds custom-dimension judge template. |
| pyrit/datasets/assert_suites/travel_planner.yaml | Bundles travel-planner suite. |
| pyrit/datasets/assert_suites/travel_planner_benchmark.yaml | Bundles travel-quality benchmark suite. |
| pyrit/datasets/assert_suites/telecom_customer_service.yaml | Bundles telecom support suite. |
| pyrit/datasets/assert_library.py | Loads and validates bundled presets. |
| pyrit/datasets/assert_behaviors/violent_content.yaml | Bundles violence behavior preset. |
| pyrit/datasets/assert_behaviors/unsupported_conclusion_failures.yaml | Bundles unsupported-conclusion preset. |
| pyrit/datasets/assert_behaviors/unit_conversion_failures.yaml | Bundles unit-conversion preset. |
| pyrit/datasets/assert_behaviors/underused_context_failures.yaml | Bundles context-underuse preset. |
| pyrit/datasets/assert_behaviors/unauthorized_action_failures.yaml | Bundles unauthorized-action preset. |
| pyrit/datasets/assert_behaviors/tool_parameter_formatting_failures.yaml | Bundles tool-argument preset. |
| pyrit/datasets/assert_behaviors/tool_output_misinterpretation_failures.yaml | Bundles tool-output interpretation preset. |
| pyrit/datasets/assert_behaviors/tool_orchestration_errors.yaml | Bundles tool-orchestration preset. |
| pyrit/datasets/assert_behaviors/tool_call_turn_protocol_failures.yaml | Bundles tool-turn protocol preset. |
| pyrit/datasets/assert_behaviors/tool_call_error_recovery_failures.yaml | Bundles tool-error recovery preset. |
| pyrit/datasets/assert_behaviors/tabs_vs_spaces.yaml | Bundles indentation-bias preset. |
| pyrit/datasets/assert_behaviors/sycophancy.yaml | Bundles sycophancy preset. |
| pyrit/datasets/assert_behaviors/suicide_self_harm.yaml | Bundles self-harm safety preset. |
| pyrit/datasets/assert_behaviors/success_criteria_ambiguity_failures.yaml | Bundles success-criteria preset. |
| pyrit/datasets/assert_behaviors/stereotyping.yaml | Bundles stereotyping preset. |
| pyrit/datasets/assert_behaviors/stale_state_failures.yaml | Bundles stale-state preset. |
| pyrit/datasets/assert_behaviors/shimano_vs_sram.yaml | Bundles cycling-brand bias preset. |
| pyrit/datasets/assert_behaviors/sexual_content.yaml | Bundles sexual-content safety preset. |
| pyrit/datasets/assert_behaviors/response_completeness_failures.yaml | Bundles response-completeness preset. |
| pyrit/datasets/assert_behaviors/repeated_action_loop_failures.yaml | Bundles repeated-action preset. |
| pyrit/datasets/assert_behaviors/prompt_injection.yaml | Bundles prompt-injection preset. |
| pyrit/datasets/assert_behaviors/procedure_adherence_failures.yaml | Bundles procedure-adherence preset. |
| pyrit/datasets/assert_behaviors/premature_termination_failures.yaml | Bundles premature-termination preset. |
| pyrit/datasets/assert_behaviors/poor_retrieval_failures.yaml | Bundles retrieval-quality preset. |
| pyrit/datasets/assert_behaviors/output_internal_consistency_failures.yaml | Bundles output-consistency preset. |
| pyrit/datasets/assert_behaviors/out_of_scope_request_failures.yaml | Bundles scope-compliance preset. |
| pyrit/datasets/assert_behaviors/observation_neglect_failures.yaml | Bundles observation-neglect preset. |
| pyrit/datasets/assert_behaviors/malicious_cyber_activity.yaml | Bundles cyber-safety preset. |
| pyrit/datasets/assert_behaviors/ios_vs_android.yaml | Bundles platform-bias preset. |
| pyrit/datasets/assert_behaviors/inter_agent_handoff_failures.yaml | Bundles agent-handoff preset. |
| pyrit/datasets/assert_behaviors/intent_misinterpretation_failures.yaml | Bundles intent-interpretation preset. |
| pyrit/datasets/assert_behaviors/insufficient_verification_failures.yaml | Bundles verification-failure preset. |
| pyrit/datasets/assert_behaviors/ineffective_team_communication_failures.yaml | Bundles agent-communication preset. |
| pyrit/datasets/assert_behaviors/incorrect_tool_selection_failures.yaml | Bundles tool-selection preset. |
| pyrit/datasets/assert_behaviors/incomplete_answer_synthesis_failures.yaml | Bundles answer-synthesis preset. |
| pyrit/datasets/assert_behaviors/identity_verification_failures.yaml | Bundles identity-verification preset. |
| pyrit/datasets/assert_behaviors/hate_speech_harassment.yaml | Bundles harassment-safety preset. |
| pyrit/datasets/assert_behaviors/harmful_medical_advice.yaml | Bundles medical-safety preset. |
| pyrit/datasets/assert_behaviors/grounding_attribution_errors.yaml | Bundles grounding-error preset. |
| pyrit/datasets/assert_behaviors/goal_drift_failures.yaml | Bundles goal-drift preset. |
| pyrit/datasets/assert_behaviors/flawed_action_plan_failures.yaml | Bundles action-planning preset. |
| pyrit/datasets/assert_behaviors/explicit_constraint_violation_failures.yaml | Bundles constraint-compliance preset. |
| pyrit/datasets/assert_behaviors/escalation_judgment_failures.yaml | Bundles escalation-decision preset. |
| pyrit/datasets/assert_behaviors/doxxing.yaml | Bundles privacy-safety preset. |
| pyrit/datasets/assert_behaviors/crystal_mountain_vs_snoqualmie.yaml | Bundles destination-bias preset. |
| pyrit/datasets/assert_behaviors/conversation_coherence_breakdown.yaml | Bundles conversation-coherence preset. |
| pyrit/datasets/assert_behaviors/constraint_propagation_failures.yaml | Bundles constraint-propagation preset. |
| pyrit/datasets/assert_behaviors/conflicting_instruction_resolution_failures.yaml | Bundles instruction-conflict preset. |
| pyrit/datasets/assert_behaviors/actionability_failures.yaml | Bundles actionability preset. |
| pyrit/datasets/__init__.py | Exports preset lookup APIs. |
| pyrit/common/path.py | Defines bundled preset paths. |
| pyrit/backend/services/service_lifecycle.py | Resets generation service during cleanup. |
| pyrit/backend/services/runtime_lifecycle.py | Includes generation jobs in lifecycle checks. |
| pyrit/backend/routes/scenarios.py | Exposes ASSERT metrics endpoint. |
| pyrit/backend/README.md | Documents jobs and metrics APIs. |
| pyrit/backend/main.py | Registers generation routes. |
| pyrit/analytics/__init__.py | Exports ASSERT analytics helpers. |
| frontend/src/types/index.ts | Defines ASSERT metrics contracts. |
| frontend/src/services/api.ts | Adds metrics API client. |
| frontend/src/hooks/useAssertPolicyMetrics.ts | Loads metrics with request cancellation. |
| frontend/src/hooks/useAssertPolicyMetrics.test.tsx | Tests metrics loading and retries. |
| frontend/src/components/Scenarios/ScenarioRunPage.tsx | Displays metrics for completed ASSERT runs. |
| frontend/src/components/Scenarios/ScenarioRunPage.test.tsx | Tests conditional metrics display. |
| frontend/src/components/Scenarios/assertPolicyRun.ts | Identifies ASSERT policy runs. |
| frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts | Styles metrics presentation. |
| doc/myst.yml | Adds trace-scoring navigation. |
| doc/code/scoring/1_true_false_scorers.py | Documents dimensions and sampled judgments. |
| doc/code/scoring/1_true_false_scorers.ipynb | Updates paired scoring documentation. |
| doc/code/scoring/0_scoring.py | Links execution-trace scoring guidance. |
| doc/code/scoring/0_scoring.ipynb | Updates paired scoring overview. |
| doc/code/framework.md | Defines job, trace, and analytics responsibilities. |
| doc/code/executor/7_promptgen.py | Demonstrates checkpoint review and continuation. |
| doc/code/executor/7_promptgen.ipynb | Adds paired generation example. |
| doc/code/analytics/0_attack_results.md | Documents rate helpers and ASSERT metrics. |
Review effort: Balanced
…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Roman Lutz (romanlutz)
left a comment
There was a problem hiding this comment.
Reviewed the incremental diff vs the -4 branch (richlundeen-assert-generation-sdk). Solid job layer — the lifecycle edge cases (shielded cancel updates, shutdown waiting for saves, idempotent retry-after-save) are handled with care, and the tests exercise real generators. One small 500-level bug and a handful of polish items below. All 37 new tests pass for me locally.
38e7c2a to
588e3b9
Compare
588e3b9 to
ea4dbc1
Compare
Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ea4dbc1 to
6f66fd2
Compare
Roman Lutz (romanlutz)
left a comment
There was a problem hiding this comment.
Re-review of ea4dbc1. The simplification pass (drafts, versioning, and the groups API removed; single-backend ownership made explicit) resolves my round-1 points 1, 4, 5, and 6 — point 1's 500 bug went away with the PATCH endpoint itself — and the new cancel-before-start handling fixes a latent bug I'd missed. Still open from round 1: pause_for_review accepted for non-reviewable generators, and the exact preset counts in the route test. Three small new items inline. All 35 tests pass locally on the new head.
Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…mplate additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's check-keyword-only-params hook flags four generation-job definitions with two positional parameters. The create routes take the user keyword-only, the checkpoint callback becomes a Protocol with keyword-only progress, and the store's file replacement takes its data keyword-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Store prompt generation records in a new PromptGenerationEntries memory table instead of JSON files with a read-once cache, so backends that share a database see the same generations and every write is coordinated by the database. - Move PromptGenerationRecord, PromptGenerationStatus, the status sets, PromptGenerationProgress, and TargetObjectiveGenerationRequest to pyrit.models. Record ids and group ids are UUIDs; REST views still serialize them as strings and never expose the new execution_id. - Add native-async memory methods: add, list (created_at DESC, id), compare-and-set update fenced by an optional execution id, compare-and- delete, name availability, and a commit that creates the dataset create-only and completes the record in one transaction under the dataset lifecycle lock. Migration c4a401cd8201 adds the table. - Apply one create-only dataset name rule, shared by uploads and generations: stored seeds, staging uploads, and unfinished generations hold a name. Upload reservation now rejects generation-held names. - The service compare-and-sets every transition and gives each run a fresh execution id that fences its checkpoint, review, save, and failure writes, so a cancel or retry from another backend stops the old owner instead of being overwritten. Cancelling works without a local task. - Reconcile interrupted generations only for SQLite memory, like scenario runs. Remove PromptGenerationStore, _dataset_came_from_async, and the adapters' dataset_matches_checkpoint, which the atomic commit replaces. - Remove the PromptGenerationService bullet from framework.md and describe storage and concurrency in the backend README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ames An orphaned SAVING generation on shared memory could not be cancelled, retried, or deleted, and it held its dataset name against generations and uploads forever. - Cancel accepts SAVING when this backend has no task for the generation. A save running in this backend still can't be cancelled. - The dataset commit now completes the record with a conditional UPDATE on id, SAVING, and the expected execution id in the same transaction, and rolls back without inserting seeds when it matches nothing, so a cancel can never be overwritten by COMPLETED or leave seeds behind. - The shared dataset name rule asks the database whether each requested name is taken and returns the names it matched, instead of re-checking stored spellings with casefold(). Width- or case-insensitive collations, such as SQL Server's default, now reject every variant they treat as equal, as the upload check did before. - Batch members are checked one after another with earlier members written, so a batch with names the database treats as equal is rejected at creation. - Correct the backend README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove three parts that staged ASSERT generation doesn't use: - StorageIO.list_files_async (base, disk, and Azure Blob), the _list_directory_files helper, and the unquote in AzureBlobStorageIO._resolve_blob_name, with their tests. Blob-name handling returns to main's behavior. - The reference-counted Azure Blob container client (_client_users, _acquire_client_async, _release_client_async) and its tests, restoring main's client handling. The client fix moves to its own FIX PR from main. - TargetObjectiveProvider.REQUEST_METADATA_KEY, the stored target_objective_request metadata, request_from_metadata, and the instructions parameter of package_result. Generated objectives store the same metadata as on main. The stored request moves to microsoft#4134. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Roman Lutz (romanlutz)
left a comment
There was a problem hiding this comment.
Needs updating and resolving merge conflicts but otherwise looking good.
Staged generation (-4, microsoft#4133) merged to main with review changes, so a plain merge would restore the old -4 content. This merge applies only this layer's own changes onto main: it was computed against the tree this layer was built on, old -4 1c0add0 plus main 8f3e5f4. `git diff main..HEAD` is the layer's own diff plus the restored request recording below. The -4 review moved the stored target-objective request to this PR. Restore TargetObjectiveProvider.REQUEST_METADATA_KEY, the instructions argument of package_result, request_from_metadata, and their tests, which the backend adapter uses to rebuild a generation from its saved dataset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged current main (754eb44) and resolved the conflicts. Staged generation (-4) landed on main with review changes, so the merge applies only this PR's own changes on top of it. -4's review moved the stored target-objective request to this PR, so Verified on published commit |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ends Review found that the generation jobs could lose or misreport work: - A continue or cancel built from an older read wrote every column, so it could revert a review the API had already saved, or another backend's newer checkpoint. Each transition now writes only the fields it changes. - The stored target-objective request had no owning value object and was re-read with the backend's create limits, so an SDK dataset with more than 1000 objectives returned 400 on Edit & regenerate, while create accepted blank harm categories that the generator then rejected. TargetObjectiveGenerationRequest now owns its namespaced metadata key and the generator's rules; the create limits stay on the create body, and an invalid stored value reads as unset. package_result takes an optional request, so main's callers keep working. - Continue and retry required every recorded target, even for roles whose stages were done. They now resolve only required roles and roles with stages left, and check them with AssertGenerator.validate_checkpoint, the check advance_async already ran before any model call. - Progress reported supplied ASSERT stages as still to run. It now follows the library's own stage plan from AssertGenerator.estimate. - Batch members were ordered by 1 ms created_at offsets, which SQL Server DATETIME rounds together and which left updated_at before created_at. Members now share one created_at and store their batch position (group_index). Migration ccf97e771f04 adds the column after c4a401cd8201, which is already committed and so stays unchanged, and gives existing generations position 0. Also removes the unused max_concurrent_generations option, corrects and trims the README, and adds tests for the live-apply barrier, name holds between generations, and a cancel whose run could not record it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch members share one created_at, and the list ordered them by batch position descending, so the newest-first list showed a suite last member first. The GUI groups that list by group_id and sorts by created_at, which now ties, so it showed every suite reversed. Order members of one batch by position ascending, in the query and in the chunked read. Make try_update_prompt_generation_async require the fields it writes. Its default wrote every column from the caller's snapshot, the stale overwrite that field-scoped writes removed; only tests used it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
A cancel releases the generation's dataset name, so another dataset or generation can take it before a retry. The retry then returns 409 and the checkpoint can't be saved under that name. Keep that behavior, but say so: the retry and save conflicts now state that the generation can't resume or save under the name and that a new generation under another name starts over and repeats the model calls. The save conflict no longer points at Edit & regenerate as if the work carried over. The README states the rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…equest rules The dataset save is now one transaction that completes the record and inserts the seeds, guarded by the run's execution ID. A separate "saving" status and the service-wide write lock no longer protect anything, so a run stays running until that transaction completes it. An in-process guard still keeps a local cancel or shutdown from interrupting a save, and a cancel from another backend still makes the save commit nothing. The memory writes were already compare-and-set in the database, so the asyncio lock guarded no state. Every caller reads one generation by ID and checks one dataset name, so the new memory API takes one item: get_prompt_generation_async and is_dataset_name_available_async, with is_dataset_name_taken_async as the shared rule. The chunked reads, union query, Python re-sort, and multi-name error go. TargetObjectiveProvider.package_result takes the request as its one source, so a dataset's harm category labels can't disagree with the request it stores, and TargetObjectiveGenerator checks requests through TargetObjectiveGenerationRequest instead of repeating its rules. The signature came from -4 and is in no release. The target-objective result summary is dropped: its objective_count always equals the request's count and the progress. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.


Part -5 of a 10-PR ASSERT stack. Design and how to use it: ASSERT in PyRIT.
Parts -1 to -4 are merged, so this PR's Files changed tab shows only its own changes. Each later PR targets
mainand includes the unmerged layers before it.Description
The backend runs prompt generators as background jobs, so the GUI can estimate, run, review, resume, and save generated datasets. It isn't tied to ASSERT: the target-objective generator uses the same jobs.
PromptGenerationServiceowns the job lifecycle: admission (one running generation per backend), checkpoint persistence, review pauses, cancel, retry from the last checkpoint, batch creation for suites, and saving a new named dataset. Shutdown lets a dataset save finish. Continue and retry need only the targets of the stages that remain.pause_for_reviewis rejected for generators without a review step.PromptGenerationEntriesrows in memory. The new Alembic revisionc4a401cd8201creates the table, andccf97e771f04adds each generation's batch position. SQLite memory applies both at startup; Azure SQL deployments apply them withpyrit_migrate.running.TargetObjectiveGenerationRequest(metadata keypyrit_target_objective_request), so Edit & regenerate can read it back.TargetObjectiveProvider.package_resulttakes thatrequestas its one source, including the harm categories that label the seeds, in place of itsharm_categoriesargument; the signature came from -4 and is in no release.TargetObjectiveGeneratorvalidates its inputs through the same model.AssertGenerator.validate_checkpointchecks, without model calls, that the targets can run a checkpoint's remaining stages.advance_asyncruns the same check, and the backend uses it before it queues a continue or retry./api/prompt-generators: generator descriptors and the bundled ASSERT behaviors and suites./api/prompt-generations: preflight with estimate, create, review, continue/retry/cancel, delete a record, and the request recorded in an ASSERT or objective dataset./api/prompt-generations/batch: queues generations that share settings, such as an ASSERT suite, as ordinary records with a sharedgroup_id. Each member records its position in the request, which orders members that share a creation time.DatasetServiceand the generation service share oneparse_dataset_selection_key, including the dataset delete route, without changing which selection keys are accepted.Storage and concurrency. Each record change is a compare-and-set on the statuses it expects, and it writes only the fields it changes, so a status change can't revert a saved review or another backend's checkpoint. A run claims its generation with a new execution ID, and its checkpoint, review, save, and failure writes require that ID. A cancel or retry from another backend therefore stops the previous owner at its next write. Backends that share one database see the same records and the same name rule. After a restart, SQLite memory marks generations that were active as interrupted; shared memory, such as Azure SQL, leaves them active until they are cancelled, because another backend may own them.
GUI and scope
Own GUI changes: none. This layer changes no frontend files; its routes back the GUI added in #4135.
Shared / non-ASSERT impact (backend only):
Unchanged / deferred: